{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "# 定义嵌入模型",
   "id": "7a70348959f05fdb"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-11-24T08:49:23.205012Z",
     "start_time": "2025-11-24T08:49:23.174005Z"
    }
   },
   "cell_type": "code",
   "source": [
    "import os\n",
    "\n",
    "import dotenv\n",
    "import langchain_community.document_loaders\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "dotenv.load_dotenv(override=True)\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.getenv(\"OPENAI_API_KEY\")\n",
    "os.environ[\"OPENAI_BASE_URL\"] = os.getenv(\"OPENAI_BASE_URL\")\n",
    "\n",
    "# 定义嵌入模型\n",
    "EMBEDDING_MODEL = OpenAIEmbeddings(model=\"text-embedding-ada-002\")"
   ],
   "id": "dac58c61451cd8fe",
   "outputs": [],
   "execution_count": 3
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 文档嵌入模型 Text Embedding Models\n",
    "## 嵌入模型概述\n",
    "`Text Embedding Models`：文档嵌入模型，提供将文本编码为向量的能力，即 **文档向量化** 。 **文档写入**和 **用户查询匹配** 前都会先执行文档嵌入编码，即向量化。\n",
    "\n",
    "LangChain从开源到专有API 提供了 **超过25种** 不同的嵌入提供商和方法的集成。\n",
    "\n",
    "`Hugging Face` 等开源社区提供了一些文本向量化模型（例如BGE），效果比闭源且调用API的向量化模型效果好，并且 **向量化模型参数量小，在CPU上即可运行**。所以，这里推荐在开发RAG应用的过程中，使用 **开源的文本向量化模型** 。此外，开源模型还可以根据应用场景下收集的数据对模型进行微调，提高模型效果。\n",
    "\n",
    "LangChain中针对向量化模型的封装提供了两种接口，一种针对 **文档的向量化(embed_documents)** ，一种针对 **句子的向量化embed_query** 。"
   ],
   "id": "e9de4dc687fd7c85"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 句子的向量化（embed_query）",
   "id": "5ead9aae33831aae"
  },
  {
   "cell_type": "code",
   "id": "initial_id",
   "metadata": {
    "collapsed": true,
    "ExecuteTime": {
     "end_time": "2025-11-24T08:49:34.464910Z",
     "start_time": "2025-11-24T08:49:33.743101Z"
    }
   },
   "source": [
    "text = \"What was the name mentioned in the conversation?\"\n",
    "\n",
    "embedding_result = EMBEDDING_MODEL.embed_query(text)    # 向量化,float列表\n",
    "\n",
    "print(len(embedding_result))\n",
    "print(embedding_result[:5])"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1536\n",
      "[0.005329647101461887, -0.0006122003542259336, 0.0389961302280426, -0.002898985054343939, -0.008904732763767242]\n"
     ]
    }
   ],
   "execution_count": 4
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "## 文档的向量化（embed_documents）\n",
    "文档的向量化，接收的参数是字符串数组。"
   ],
   "id": "3db798b78bb39f9e"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "### 举例一、文本列表向量化",
   "id": "550199af88bde7e9"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-11-24T08:54:13.155691Z",
     "start_time": "2025-11-24T08:54:11.891394Z"
    }
   },
   "cell_type": "code",
   "source": [
    "texts = [\n",
    "    \"What was the name mentioned in the conversation?\",\n",
    "    \"The name was John Doe.\",\n",
    "    \"Jane Doe was also mentioned.\"\n",
    "]\n",
    "\n",
    "embedding_results = EMBEDDING_MODEL.embed_documents(texts)\n",
    "\n",
    "print(len(embedding_results))\n",
    "for i in range(len(embedding_results)):\n",
    "    print(len(embedding_results[i]))\n",
    "    print(f\"embedding_results[i][:3]\", end=\"\\n\\n\")"
   ],
   "id": "14dd1a8c6f6088a0",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n",
      "1536\n",
      "embedding_results[i][:3]\n",
      "\n",
      "1536\n",
      "embedding_results[i][:3]\n",
      "\n",
      "1536\n",
      "embedding_results[i][:3]\n",
      "\n"
     ]
    }
   ],
   "execution_count": 5
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 举例二、加载文档并向量化",
   "id": "3ca521d64b5a7110"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-11-24T09:05:55.489978Z",
     "start_time": "2025-11-24T09:05:54.584403Z"
    }
   },
   "cell_type": "code",
   "source": [
    "from langchain_community.document_loaders import CSVLoader\n",
    "\n",
    "loader = CSVLoader(\"asset/load/03-load.csv\")\n",
    "docs = loader.load()\n",
    "\n",
    "# 存放的是每一个chunk的embedding\n",
    "embedding_results = EMBEDDING_MODEL.embed_documents([doc.page_content for doc in docs])\n",
    "\n",
    "print(len(embedding_results))\n",
    "for i in range(len(embedding_results)):\n",
    "    print(len(embedding_results[i]))\n",
    "    print(f\"embedding_results[i][:3]\", end=\"\\n\\n\")"
   ],
   "id": "b3a8342e53692f32",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "1536\n",
      "embedding_results[i][:3]\n",
      "\n",
      "1536\n",
      "embedding_results[i][:3]\n",
      "\n",
      "1536\n",
      "embedding_results[i][:3]\n",
      "\n",
      "1536\n",
      "embedding_results[i][:3]\n",
      "\n"
     ]
    }
   ],
   "execution_count": 6
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
